Add all needed targets to perform airgap installs on GKE by viccuad · Pull Request #317 · SUSE/catapult

viccuad · 2020-11-05T10:56:24Z

This PR enables testing airgapped clusters on GKE. It contains targets for setting up a
local-registry, pushing the images from imagelist.txt to the registry, and airgapping
the cluster via k8s networkPolicies. With this, one can test airgapped installs
of kubecf & quarks-operator.

Their implementation has been consciously made to be BACKEND-independent.

Prerequisites for full airgap installations:

k8s networking plugin installed (for now, only GKE cluster).
Insecure container registries in k8s internal network accepted (for now, only
GKE cluster).

Included in this PR:

Add BACKEND-side targets airgap-up and airgap-down for GKE and stubs for
the rest of backends. Those targets can be expanded in the future when each
backend satisfies the prerequisites.
Add networkPolicies for isolating the cluster by only allowing network
connections in the internal network, and external connections on DNS ports,
for anything on the default, scf, and cf-operator namespaces.
These network policies are backend-independent, but they need a k8s Network
plugin present. At the moment, only GKE gets deployed with such, in that case,
Calico (by consuming Add Calico as networking plugin to GKE cap-terraform#98).
Add a module-experimental-local-registry target. This target deploys a
Docker v2 registry as a service in ns default, only exposing it internally
to the cluster via the ClusterIP and the default
local-registry.default.svc.cluster.local DNS name. For Kubelet to correctly
pull images from the local registry, one needs to target its ClusterIP.
Add a module-experimental-push-imagelist target. This target copies all
images listed in any imagelist.txt file in the buildfolder into the local
registry. This is done by spawning 1 k8s job per image to copy, each of which
performs a skopeo copy and waiting for all jobs to finish successfully.
We are using dragonchaser/opensuse-skopeo:latest image, which contains
podman and skopeo.
Add a module-experimental-podman-imagelist target. This target spawns a pod
with podman configured with vfs for unprivileged containers, and the
imagelist.txt files and a testing script inside the /tmp/ folder. Only useful
for manual testing and developing the airgap.

This deploys a Docker Registry v2 image. Registry is exposed in local-registry.default.svc.cluster.local and it is set up with a pvc. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

This target parses all imagelist.txt files inside the buildfolder. For each image listed in the files, it spawns a kube job that uses `skopeo copy` to copy the image to the local repository. The target waits for all jobs to have succeeded. We are using dragonchaser/opensuse-skopeo:latest image, which contains podman (with vfs for unpriviliged containers), and skopeo. The kube jobs are configured to retry for 4 times, in case the jobs fail (they may, as all the jobs start at the same time, and my clog the cluster). Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

This is just a test target with a podman image that allows us to do podman pull. Signed-off-by: Christian Richter <crichter@suse.com>

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

Also, don't call airgap-down and then airgap-up, when calling the old airgap-up target.

mook-as · 2020-11-20T20:02:53Z

+
+.PHONY: airgap-up airgap-down
+airgap-up airgap-down:
+	echo "Not implemented yet" && exit 1


That's basically moving module-experimental-airgap-up here right?
I'm okay with doing that in a follow-up PR, but I expect by the time this whole thing is done we'll drop the experimental version?

Shouldn't the top-level makefile have a

airgap-up airgap-down: $(MAKE) -C backend/$(BACKEND) $@

mook-as · 2020-11-20T20:08:20Z

+. ../../include/common.sh
+. .envrc
+
+export KUBECF_NAMESPACE=scf


It would be good to make this configurable (mostly because currently the namespace on GitHub CI is kubecf, because it's just using make kubecf-apply).

Of course, this will all change once RFD9 is in place.

mook-as · 2020-11-20T20:13:00Z

+. ../../include/common.sh
+. .envrc
+
+export KUBECF_NAMESPACE=scf


mook-as · 2020-11-20T20:15:49Z

+export KUBECF_NAMESPACE=scf
+export QUARKS_NAMESPACE=cf-operator
+for ns in $KUBECF_NAMESPACE $QUARKS_NAMESPACE default; do
+    kubectl create namespace $ns || true


kubectl create namespace "${ns}" --dry-run=client -o yaml | kubectl apply -f -

… now that I've typed it out that's just overly verbose and doesn't really help much, does it? I guess it catches errors other than the namespace already existing, but then we'll fail when we label it anyway. Nevermind, then.

mook-as · 2020-11-20T20:46:44Z

+      # GKE: Uses Calico, Use `--enable-network-policy` when
+      # creating a cluster (`gcloud`).
+      # Minikube needs special setup.
+      # KinD used by our Drone setup may have support.


FWIW, I don't think we have plans for a drone setup any longer.

mook-as · 2020-11-20T21:42:22Z

+        MIRROR='local-registry.default.svc.cluster.local'
+        echo ">>>>> Mirroring image: ${SOURCE_IMAGE}"
+        # add registry.suse.com/cap/ in front if it's not already there
+        if [[ $( echo $SOURCE_IMAGE | grep -o '/' | tr -d '\r\n' | wc -c) -lt 1 ]]; then


I'm trying to parse this line.

Source $SOURCE_IMAGE grep tr wc test

Docker library debian:latest (empty) (empty) 0 true

Docker Hub cfcontainerization/image:tag / / 1 false

Registry registry.suse.com/cap/image:tag /␤/ // 2 false

So this will only mirror the "library" images (like debian:latest)? If that's the case, I think this line is equivalent to:

Suggested change

if [[ $( echo $SOURCE_IMAGE | grep -o '/' | tr -d '\r\n' | wc -c) -lt 1 ]]; then

if [[ -z "$(tr --complement --delete / <<< "${SOURCE_IMAGE}")" ]]; then

If we actually wanted to do this for zero-or-one-slash things, that would be:

Suggested change

if [[ $( echo $SOURCE_IMAGE | grep -o '/' | tr -d '\r\n' | wc -c) -lt 1 ]]; then

if [[ "$(tr --complement --delete / <<< "${SOURCE_IMAGE}")" -lt 2 ]]; then

mook-as · 2020-11-20T22:18:02Z

+
+while IFS= read -r file
+do
+    for SOURCE_IMAGE in $(cat "$imagelist_file"); do


Is this supposed to refer to file on line 42 instead?

mook-as · 2020-11-21T00:07:26Z

+    spec:
+      containers:
+      - name: copy-image
+        image: dragonchaser/opensuse-skopeo:latest


This image doesn't have the labels to indicate where the source comes from; we should probably add that. I assume that's modules/experimental/docker/Dockerfile.skopeo though?

mook-as · 2020-11-21T00:24:51Z

+
+.PHONY: podman-imagelist
+podman-imagelist:
+	./podman-imagelist.sh


Hmm, it's unclear to me why we have two versions.

mook-as · 2020-11-21T00:25:44Z

+  # Include insecure registry
+  # shellcheck disable=SC2087
+  ssh -T sles@${kube_node} << EOF
+sudo sed 's/^\(CRIO_OPTIONS\s*=\s*\).*$/\1"--insecure-registry=${DOCKER_REGISTRY}"/' \


We're wiping any existing options?

thardeck and others added 8 commits November 5, 2020 10:40

Add module-experimental-local-registry target

614083e

This deploys a Docker Registry v2 image. Registry is exposed in local-registry.default.svc.cluster.local and it is set up with a pvc. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

Move isolate_network() to gke_isolate_network() in include/func.sh

9ab51ad

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

Add airgap-up and airgap-down targets on GKE backend

7b4fd33

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

Add no-ops for airgap-up and airgap-down targets on rest of backends

44cda0f

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

Add a module-experimental-podman-imagelist

92be772

This is just a test target with a podman image that allows us to do podman pull. Signed-off-by: Christian Richter <crichter@suse.com>

Add Dockerfile.skopeo for module-experimental-push-imagelist target

c6bd718

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

Configure insecure registry on old airgap target for CaaSP

0b15fd6

Also, don't call airgap-down and then airgap-up, when calling the old airgap-up target.

viccuad requested review from dragonchaser, jimmykarily, prabalsharma and thardeck November 5, 2020 10:56

This was referenced Nov 5, 2020

Internet-less suite for GH Actions cloudfoundry-incubator/kubecf#1148

Open

Use weave in CI so we can run the internetless suite cloudfoundry-incubator/kubecf#671

Closed

viccuad mentioned this pull request Nov 20, 2020

feat: CI: GHA: brains, internetless, and CCDB rotation cloudfoundry-incubator/kubecf#1610

Merged

7 tasks

mook-as requested changes Nov 21, 2020

View reviewed changes

Navigation Menu

Uh oh!

Add all needed targets to perform airgap installs on GKE - #317

Add all needed targets to perform airgap installs on GKE#317
viccuad wants to merge 8 commits into
masterfrom
viccuad/airgap

viccuad commented Nov 5, 2020 •

edited

Loading

Uh oh!

mook-as Nov 20, 2020

Uh oh!

mook-as Nov 20, 2020

Uh oh!

mook-as Nov 20, 2020

Uh oh!

mook-as Nov 20, 2020

Uh oh!

mook-as Nov 20, 2020

Uh oh!

mook-as Nov 20, 2020

Uh oh!

mook-as Nov 20, 2020

Uh oh!

mook-as Nov 21, 2020

Uh oh!

mook-as Nov 21, 2020

Uh oh!

mook-as Nov 21, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Source	`$SOURCE_IMAGE`	`grep`	`tr`	`wc`	test
Docker library	`debian:latest`	(empty)	(empty)	0	true
Docker Hub	`cfcontainerization/image:tag`	`/`	`/`	1	false
Registry	`registry.suse.com/cap/image:tag`	`/␤/`	`//`	2	false

	if [[ $( echo $SOURCE_IMAGE \| grep -o '/' \| tr -d '\r\n' \| wc -c) -lt 1 ]]; then
	if [[ -z "$(tr --complement --delete / <<< "${SOURCE_IMAGE}")" ]]; then

	if [[ $( echo $SOURCE_IMAGE \| grep -o '/' \| tr -d '\r\n' \| wc -c) -lt 1 ]]; then
	if [[ "$(tr --complement --delete / <<< "${SOURCE_IMAGE}")" -lt 2 ]]; then

Uh oh!

Conversation

viccuad commented Nov 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

viccuad commented Nov 5, 2020 •

edited

Loading